NAVAL 

POSTGRADUATE 

SCHOOL 


MONTEREY,  CALIFORNIA 


THESIS 


ENHANCEMENT  OF  THE  ACQUISITION  PROCESS  FOR 
A  COMBAT  SY  STEM— A  CASE  STUDY  TO  MODEL  THE 
WORKFLOW  PROCESSES  FOR  AN  AIR  DEFENSE 
SYSTEM  ACQUISITION 

by 

Wpc  Lee  Chia 
t>eccmbcr  2009 

Thesis  Advisor:  James  Bret  Michael 

Thesis  Cc'Advisor:  Man*Tak  Sbing 


Appro  od  for  public  release:  distribudoo  Is  unUmiied 


REMRT  DOC  t MENTATION  PACE 


Fom  ApprovtiJ  0S4B  No  07nt-mM 


^•Mic  Tcpe*<ifi^  But4eo  for  Uiis  coUcbMo  of  uifonmiion  i<  to  avojge  I  liour  per  retponte,  oichidittg  Uw  ome  for  re'iCMtt^  imuctMtv 

tearohifi^  euftiog  4ab  gedtctiog  and  luioianui^  die  deia  nMded,  and  C0119ICI104  and  rcpiewni  die  coUecsoo  of  uifonation  Send 

aioiasiB  iggifAng  diis  biMeit  eennte  e*  wy  otb9  aapeci  of  tius  coSlection  of  irtfbrnMtoiv  irteliadiei  tuggefions  Eoc  reduciog  diis  Mrdeiv  10 
Wodvopon  heedaianea  Steven,  Diiectonte  fee  Inforrafion  OpetViont  aod  R^oeti,  1 2 1 S  JelTenon  Ooi  i<  I  bgliovy,  Suite  1204,  ArUogton,  VA 
12202'43<12,sidio  die  Office  of  Wwie|ooiei«  aM  Oitffee  Peperweet  Redwneo  Prefect  (()’Ci4«0)g4l|  Wwhn^tai  DC20S03 


I.  AC.ENCV  use  ONLY  tU*^  MmA;  2.  REPORT  DATE  3.  REPMT  TYTE  AND  DATES  COVERED 


Decemter  2009 


A  TITLE  AND  SUBTITLE  Ettlafintiteni  of  (he  Aoiuiuiiott  Proctn  tor  a  ConMl 
Sysiem — A  Case  Siudy  to  Motkl  ilie  Workflow  Procc^wi  for  an  Air  Defenw  System 
AcquRiOon 


AALTHORfSl  WnLeeCTtia 


r  PERPORMINR  ORGANIZATION  NAME<S)  AND  ADDRESSfES) 
Naval  Poaigradiiair  School 
Monirrcy.CA  93943-iOOO 


9.  SPONSORING  /MONITORING  AGENCY  NAME^Si  AND  ADDRESStCS) 
N/A 


Master’s  'Hiaaia 


&  FUNDING  NUMBERS 


A  PERFORMING  ORGANIZATION 
REPORT  NUMBER 


10.  SPONSORINCwMOMTORING 
AGENCY  REPORT  NUMBER 


II.  SUPPLEMENTARY  NOTES  lilt  viesvsetpressM  id  titia  ihrais  arc  titosc  of  the  author  aod  <lo  rtot  reflcci  tlic  olTicia)  Mlicv 
or  DMitiott  of  tiK  Deoanmeot  of  Delcittc  or  rhe  U  S.  Ooscrnrticw. 


12a.  DISTRIBUTION  I A VaI  LABILm'  STATEMENT  12b.  DISTRIBUTION  C<H>E 

AoMovrd  for  ptiblic  release,  diairilxiiioii  is  tstlitniiod 


IL  AB.^RACT  laaalaua  100  •ardri 

This  ihesis  exammes  the  challenges  involved  in  (he  concept^rerinemenl  phase  of  (he  acquisition  process  of  a 
complex  system  We  inve«Ogaled  (he  technical  feasibility  of  using  the  Goal  Question  Metnc  meihodulogy  as  paji  of 
the  concept^reHnement  phase  of  (he  rex^uirements  analysia.  Use  case  analysis  aivd  activity  diagram  modeling  were 
used  to  artalyze  the  worYllow  of  a  generic  concept^refirtenent  process.  Statechajt  sssenioas  and  runtime  execution 
munitnnng  were  then  used  to  formally  specify  the  process  and  check  for  compliartce  to  the  process  during  its 
enactment 


lA  SU&IECT  TERMS  Acquisition  Process,  Concept-Refinement  Phase,  OQM  Method.  Use 
Case  Analysis,  Activity  Diagram.  Workflow  Process.  StateChan  Assertions.  Runtime 
Execution  Morutonng 


11.  SECL'RITV 
rL,V5SIFirATION  OF 
REPORT 

UrKlasaiflcd 


N5N  7140.0)  •ISIhSSOO 


IE  SECURITY 
CLASSIFICATION  OF  THIS 
PAGE 

linclarsincd 


!♦.  SECURITY 
CLASSIFICATION  OF 
ABSTRACT 

IJtKiarsificd 


IS.  NUMBER  OF 
PAGES 

67 


lA  PRICE  CODE 


20.  LIMITATION  or 
ABSTRACT 


Swdard  Forei  2«S  fKev  2.99) 
PresenbM  by  ANSI  Std  239.18 


THIS  PAGE  INTENTIONALLY  LEFT  BLANK 


Approved  for  public  release;  dislrlbudoo  i»  unlimited 


ENHANCEMENT  OF  THE  ACQUISITION  PROCESS  FOR  A  COMBAT 
SYSTEM— A  CASE  STUDY  TO  MODEL  THE  WORKFLOW  PROCESSES  FOR 
AN  AIR  DEFENSE  SYSTEM  ACQUISITION 

Wbc  Leo  Chifl 

Liculciunl  ColoncL  Republic  of  Singapore  Air  Force  (RSAF) 

B.Sc.  (Hons)  Comp  Sci,  The  Queen’s  University  of  Belfast,  U.K.,  1997 


Submitted  in  partial  I'ulfillment  of  the 
requirements  for  the  degree  of 


MASTER  OF  SCIENCE  IN  COMPUTER  SCIENCE 

from  the 


NAVAL  POSTGRADUATE  SCHOOL 
December  2009 


Author 


Wee  Lee  Chia 


Approved  by:  Profciior  James  Bret  Michael 

Thesis  Advisor 


Professor  Man*Tak  Shing 
Co*Advi8or 


Professor  Peter  Denning 

Chairman,  Department  ofCon^utcr  Science 


THIS  PAGE  INTENTIONALLY  LEFT  BLANK 


IV 


ABSTRACT 


This  thesis  examines  the  challenges  Involved  in  the  concept^refinement  phase  of  Ihc 
acquisition  process  of  a  complex  system.  We  investigated  the  technical  feasibility  of 
using  the  Goal  Question  Metric  methodology  as  part  of  the  concepUrctinement  phase  of 
the  requirements  analysis  Use  case  analysis  and  activity  diagram  modeling  were  used  to 
analyze  the  workflow  of  a  generic  concept*relinement  process.  Statechan  assertions  and 
runtime  execution  monitoring  were  then  used  to  formally  specify  the  process  and  check 
for  compliance  to  the  process  during  its  enactment 
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I. 


INTRODUCTION 


A.  BACKGROUND 

In  ihe  acquisition  process  of  a  complex  system,  the  goal  is  to  have  a  process  that 
produces  defensible  decisions  supported  by  sound  analyses  and  clear  rationale  With  ever 
stretched  defense  budgets  and  the  push  in  the  U.S.  for  reform  in  defense  acquisition  in  the 
form  of  minimizing  the  number  of  requirements  to  be  specified  by  military  specifications, 
the  design  trade  space  has  to  be  broadened  enough  for  examination  of  a  full  range  of 
alternatives  by  the  developing  contractor.  As  a  result,  the  prc*acquisinon  phase.  In  which 
system  concepts  are  refined  and  technology  maturity  and  capabilities  are  identified,  plays 
a  vital  role  in  the  successful  acquisition  of  today's  complex  systems.  This  phase,  also 
known  as  the  conccpUrctinemcnt  phase  in  the  U.S.  DoD  acquisition  process,'  preludes 
the  actual  initiation  of  the  program. 

Project  Managers  (PM)  need  to  consider  system  complexify,  the  aggregation  of 
emerging  and  mature  technologies,  system  Interoperability,  and  system  evolution. 
Besides  these  operational  considerations,  the  project  team  must  also  consider 
organizational  Issues.  Some  of  these  issues  will  differ  from  one  acquisition  program  to 
another,  but  there  are  issues  that  these  programs  share,  such  as  those  associated  with  the 
assessment  of  nsk  and  cost*bcnefit  analyses,  as  well  as  acceptance  and  evaluation.  Other 
considerations  include  the  type  of  acquisition  approach  to  adopt  (for  example, 
incremental,  evolutionary  or  agile),  external  interfaces,  use  of  previously  developed  or 
commercial  sofhvare,  competition/sohcltation  approach,  contracting  approach, 
information  assurance  strategy,  training,  and  support.-  The  challenges  encountered  during 
pre-acquisition  can  go  beyond  requirements  engineering,  and  operahons  to  include 
development  strategy  and  sustainment.  A  lot  has  to  do  with  human*in*thc*loop  processes 


'  Departmem  of  Defense  {DoD),  InsirucUon  $000.2,  "Opecalion  of  the  Detenw  Acquisiiion  System." 
May  12,2003 

^  Cajneg)e>Mellon  Univeiaify,  "Acqojsition  Overview:  The  Challen^s,"  hltpsr.'/buildsexurityin.ua* 
c«n.gov/dai&y>'b«i>'anicles/hesl>praclJces^acqui8ilJon/S93>BSI.Kii7il  (accessed  December  2009). 
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and  al  limes,  lo  a  certain  exicnl,  venturing  into  'unehartered  lemlory’.  In  contrast,  the 
other  phases  of  the  acquisition  process  are  typically  better  developed  as  more  research  is 
conducted  and  experience  gained  for  such  processes.  As  an  example,  in  a  system 
development  phase,  developers  or  PM  can  depend  on  an  established  model  such  as  the 
Capability  Maturity  Model  Integration  (CMMI).^  The  model  covers  the  acquisition 
discipline  as  it  relates  to  product  development  and  maintenance,  which  includes  the  use 
of  commercial  off*thc*shelf  (COTS)  products  and  outsourcing  As  a  result,  developers 
and  PM  tend  to  be  pre*disposed  to  use  wcll*cstablished  models,  with  the  perception  that 
ihc  challenges  faced  would  not  be  so  varied  and  critical  than  when  using  non*siandard 
model.  However,  the  same  cannot  be  said  for  pre^ystem  acquisinon  as  not  much  formal 
work  has  been  conducted  in  this  area.  To  address  this  gap  and  lo  meet  the  challenges, 
process  modeling  and  methodology  for  formally  capturing  acquisition  processes  must  be 
in  place 

The  objective  of  this  thesis  was  lo  investigate  ways  to  improve  the  concept* 
refinement  tasks  in  the  pre*sysiem  acquisition  process  The  scope  of  the  research  is  lo 
propo.sc  a  methodology  that  can  be  u.sed  to  derive  measurable  requirements  for  an 
acquisition  program  to  define  the  success  of  the  system  acquisition,  and  to  apply  existing 
techniques  lo  a.'uure  that  the  methodology  is  carried  out  as  specified 

In  the  acquisition  of  sofhvaie*intensive  systems,  how  can  stakeholders  specify 
their  program  requirements  in  a  non*ambiguous  way  lo  system  designers'^  More  often 
than  not.  the  vagueness  and  ambiguity  of  natural 'language  specifications  of  requirements 
can  be  easily  misinterpreted  by  Ihc  engineers  developing  a  system.  Consequently,  the 
actual  requirements  performed  may  differ  from  that  of  the  intent  of  the  stakeholders.  On 
the  other  hand,  if  engineers  represent  requirements  in  a  formal  (that  is,  mathematical) 
language  with  a  restricted  vocabulary,  stakeholders  may  not  fully  appreciate  the  logic  of 
the  specifications  due  to  lack  of  training  or  knowledge  of  formal  methods.  Hence,  a 
‘bridging  of  minds'  between  stakeholders  and  developers  is  desired  especially  in  the 
development  of  complex  systems. 

^  Sot^are  £n^ine«nng  Institute,  C£jneg)e>Mellon  Llniveisily,  "Capability  Maluciry  Model  Inlegcaliun 
http  /Aw>w.sei.cmu.edu>'cmfnv«t^fa(yre]aied-taq.cfni  (accessed  December  S,  2009) 
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As  a  step  toward  accomplishing  the  bridging,  the  author  investigated  the  well* 
established  Goal,  Question,  Metrics  (GQM)  method  by  applying  it  to  the  concept* 
refinement  phase  of  the  acquisition  of  a  combat  system,  known  as  the  Con^)lementary 
Low*Alhtude  Weapon  System  (CLAW)  for  low*aUinjde  air  defense  capability. 

The  analysis  of  CLAW  involved  creating  use  cases  and  modeling  the  workflow  of 
the  concept*reflnement  phase  with  activity  diagrams.  The  author  then  uses  these  aitifacLs 
to  guide  the  formalization  of  the  workflow  requirements  and  the  author  settled  on  using 
Staiechait  assertions  The  author  then  demonstrated  the  technical  feasibility  of  exploying 
assertion*based  validahon  and  executable  runtime  monitoring  of  the  workflow,  with  the 
aim  of  ensuring  the  correct  enactment  of  the  process. 

B.  ORGANIZATION 

The  organization  of  this  thesis  includes  an  introduction,  three  development 
chapters,  and  a  final  chapter  for  conclusions  and  tiiiure  work.  Chapter  II  presents  an 
overview  of  the  acquisition  process,  along  with  a  discussion  of  the  need  for  precise 
workflow  process  modeling.  In  Ch^tcr  HI,  the  author  introduces  the  GQM  method  as  a 
means  to  develop  measurable  program  requirements  and  provides  details  of  this  process 
through  the  en^loyment  of  a  methodology  using  a  case  study  for  the  acquLsition  system 
Chapter  IV  describes  the  modeling  of  an  acquisition  process  in  terms  of  Lfnified 
Modeling  Language  (UML)  activity  diagrams  and  the  use  of  Statechan  assertions  to 
check  for  the  proper  and  timely  execution  of  the  acquisition  tasks  in  the  acquisition 
process  model.  Chapter  V  contains  conclusions  and  topics  for  future  work. 
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II.  THE  ACQUISITION  PROCESS 


A.  OVERVIEW 

One  of  the  key  purposes  of  DoDI  5000.2  is  to  “establish  a  simplified  and  flexible 
management  framework  for  translating  mission  needs  and  teehnology  opportunities, 
based  on  approved  mission  needs  and  requirements,  into  stable,  aflbrdable,  and  well* 
managed  aequisilion  programs  that  include  weapon  systems  and  automated  information 
systems  (AISs)  It  defines  three  key  processes  that  must  work  In  concert  to  deliver  the 
capabilities  required  by  the  warfighters  the  requirement  process  (Joint  Capabilities 
Integration  &  Development  System  (JCIDS)),  the  acquisition  process  (Defense 
Acquisition  System),  and  program  and  budget  development  (Planning.  Programming, 
Budgeting,  and  Execution  [PPBE]  process)  ^ 

This  chapter  contains  an  overview  of  the  E^ctbnsc  Acquisition  Management 
Framework,  followed  by  a  review  of  process*modeling  approaches  identified  via  a  search 
of  the  open  literature.  The  intent  is  not  to  dive  Into  the  details  of  the  steps  In  the  process 
but  to  provide  a  macro  view  of  the  mechanisms  involved. 

B.  TH  E  D  E  FE  NS  E  ACQUISI  Tl  ON  M  AN AGEM  ENT  FRAM  EWORK 

The  U.S.  defense  acquisition  process  is  structured  Into  discrete  phases  separated 
by  major  decision  poinLs  (called  milestones  or  decision  reviews)  with  a  number  of  key 
activities  to  provide  the  basis  for  comprehensive  management  and  informed  decision* 
making  ^  This  is  based  on  the  policy  of  ^ 

] .  Flexibility  •  tailoring  program  strategies  and  oversight  to  fit  the  particular 
conditions  of  the  program 

*  DoD,  insimebon  3000  2 

^  Defense  Acquisition  University  {DAU),  "Inlegrated  Defense  AcquisiOon,  Technolo^,  and  LogisUcs 
Life  Cycle  Management  System,’*  June  13, 2009,  hitpsV^acc  dau  millFC/indes  hem  (accessed  Novenhec 
1 1 . 2009). 

^rbid. 

^  DoD,  Direclike  3000  I,  "The  Defense  Acquisition  System,'*  May  12, 2003 
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2.  Rf^onsiveness  •  rapid  uiicgrarion  of  advaDced  techrologics  ihiough 
evolutionary  acquisition 

3  (nnovauon  •  adoptiun  of  practices  that  reduce  cycle  lime  and  cost,  as  well 
as  CQCourage  teamwork 

4  Disciplioe  •  use  of  approved  program  baselioe  paramcLers  as  control 
objectiv'es,  identifying  deviations,  and  exit  criieria 

5  Streamlined  and  effective  management  -  project  members  are  empowered 
with  sufficient  authority  while  maintaining  accountability 


MMSOita  4,  a  c^C 

SHiru«Na«r>i  AtfuviM*  (j  SHott  s«d  h  pmi 


IOC 


FOC 


c«(ic^ 

teflnswnl 


Tvdinology 

Systotn  Ovvtfopmstt 

Ptoflucflon  S 

Oe^^apniBtTt 

a  Oawonaifrton 

O^toytTMAl 

S«JMM>t 

UWftOTSE  ^  CSmv 

Pf««SfManift  ScqiasMon  SyMmi  teqidalSon  Susainnonl 

Figure  I.  The  Defense  Acquisition  Management  Framework*^ 


The  acquisition  process  begins  with  the  identification  of  a  capability  need  that 
requires  a  material  solution.  The  process  encompasses  the  activities  of  desigti, 
fabrication,  lest,  manufacturing,  operations,  and  support  It  may  involve  modifications, 
and  it  ends  with  disposaL/recycling/dcmilitanzation  Major  upgrade  or  modification 
programs  may  also  follow  the  acquisition  life  cycle  process.^ 

There  are  Eve  major  phases  in  this  process. namely  concept  refinement, 
technology  development,  system  development  and  demonstration,  production  and 
deployment,  as  well  as  operations  and  support.  Within  Ibis  process,  ihe  finl  two  arc  part 
of  the  prc*acquisition  phase.  This  is  because  it  is  only  after  phase  two  that  the  new 
acquisition  program  is  initiated. 


^  Depsmueni  of  Defense  {DoD),  InstrucUon  5000.2,  'Operation  of  ibe  Defemo  Acquisiiion 
May  12,2003. 

**  DAU,  "bilejTated  Defense  AcquisiUon." 

DoD,  In&lnjcQon  $000  2 
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The  firel  phase  is  concept  rcfincmenl.  The  purpose  is  lo  refine  the  initial  concept 
that  was  provided  as  part  of  the  input  into  this  phase.  The  other  inputs  that  are  typically 
accompanied  by  this  process  are  the  Analysis  of  Alternatives  (AoA)  plan,  exit  criteria  as 
well  as  the  alternative  maintenance  and  sustainment  concepts.  It  is  a  very  rigorous 
process  as  the  team  has  to  analyze  the  operational  capabilities  and  environmental 
constraints,  perform  trade-off  studies  to  assess  critical  technologies  associated  with  the 
operational  concepts,  and  develop  exit  criteria  for  the  various  phases,  as  well  as  the  test 
plans.  The  phase  ends  when  the  Technology  Development  Strategy  (TDS)  document  Is 
developed  and  the  Milestone  Decision  Authority  (MDA)  approves  the  preferred  solution 
presented  In  the  TDS  document.  In  Chapter  III.  a  GQM  template  is  applied  as  an 
approach  to  develop  some  of  these  decisions 

Technology  development  Is  the  second  phase,  and  the  purpose  Is  to  reduce  the 
technological  nsk  of  the  program  by  determining  the  appropriate  set  of  technologies  to  be 
integrated  Into  a  full  system  It  is  an  iterative  process  to  assess  the  viability  of  the  various 
technologies  as  spelled  out  in  the  TDS  document.  As  such,  it  is  inevitable  that  there  Is 
close  collaboration  among  the  developers,  the  science,  engineering  and  technology 
community,  as  well  as  the  user  during  this  phase.  As  there  are  various  technologies  to  be 
considered  for  different  applications  In  a  system,  it  Is  possible  to  obtain  Incremental 
approval  by  the  MDA  once  a  proposed  solution  has  demonstrated  itself  to  be  militarily 
useful.  The  phase  ends  when  all  the  increments  of  the  militarily  useful  capability  have 
been  identified  and  successfully  demonstrated.  The  phase  can  also  end  If  the  MDA 
terminates  the  program  If  approval  is  given  by  the  MDA  lo  proceed  with  program 
initiation,  the  Capability  Development  E>ocument  (CDD),  which  spells  out  the  details  on 
operational  performance  to  design  the  proposed  system,  will  be  provided  as  an  input  for 
the  next  phase. 

The  system  develt^ment  and  demonstration  phase  marks  the  initiation  of  the 
acquisition  program.  While  the  system  is  being  developed  in  accordance  with  the  CDD, 
the  emphasis  Is  to  ensure  operahonal  supponablhty  and  minimize  the  logistics  footprint. 
Other  Actors  that  are  considered  in  this  phase  Include  ensuring  affordability  and 
protection  of  cnlleal  program  Information  and  demonstrating  system  integration, 
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Inler operability,  and  safely.  To  obtain  approval  for  the  MDA  lo  commit  Into  the  program, 
one  of  the  focal  points  is  system  demon^trahon.  This  effort  is  conducted  "when  a  system 
is  demonstrated  in  its  intended  environment,  using  the  selected  prototype;  meets 
approved  requirements;  industrial  capabilities  are  reasonably  available,  and  the  system 
meets  or  exceeds  exit  criteria."*  * 

The  purpose  of  the  production  and  deployment  phase  is  to  “achieve  an  operational 
capability  that  satisfies  mission  nceds."*^  To  ensure  this,  the  system  Is  typically  subjected 
to  numerous  operational  test  and  evaluation  (OT&E)  activities  to  determine  the 
effectiveness  and  suitability  of  the  system  prior  to  the  commencement  of  this  phase.  As  a 
result  of  this  effort,  low*rate  Initial  production  will  commence 

to  ensure  adequate  and  efficient  manu^ctunng  capability  and  to  produce 
the  minimum  quantity  necessary  to  provide  production  or  production* 
representative  articles  for  (OT&E,  establish  an  initial  produchon  base  for 
the  system;  and  permit  an  orderly  increase  in  the  production  rate  for  the 
system,  sufficient  to  lead  lo  ful Urate  production  upon  successful 
completion  of  operational  (and  hve*flre,  where  applicable)  testing 

Additionally,  llfe'cycle  support  issues,  such  as  saining,  maintenance  and  support, 
and  upgrades  should  be  re-examined  and  updated  accordingly.  One  key  outcome  arising 
Irom  this  phase  is  (he  declaration  on  (he  attainment  of  Initial  Operational  Capability 
(IOC)  by  the  system  user. 

The  fifth  phase,  also  known  as  the  sustainment  phase,  addresses  operations  and 
support  The  purpose  Is  to  establish  suf^orl  to  the  system,  which  entails  Issues  such  as 
maintenance,  upgrades,  training,  and  technical  si^port.  This  Is  pcribtmed  throughout  the 
life  cycle  of  the  system  so  that  it  can  be  managed  In  the  most  cos(*efficienl  manner  while 
maintaining  the  system's  operational  needs  Key  decisions  that  arc  made  during  this 
phase  address  the  affordability  of  the  mixing  of  systems,  as  predictions  about  cosls 
become  reality,  (n  addition,  as  threats  change  it  Is  possible  that  users  will  move  lo  retire 
certain  systems  and  procure  additional  numbers  of  other  existing  systems 

' '  DoD,  InsirucOon  SOOO  2 
Ibid. 

'^rbid. 
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What  is  key  in  this  framework  is  the  provision  of  the  necessary  documents  as 
inputs  from  one  pha.se  to  another,  for  example,  the  AoA  plan  and  Initial  Capability 
Document  (ICD),  as  inputs  to  the  conccpl*rclinemenl  phase,  before  the  phase  can  begin. 
Additionally,  the  approval  required  by  a  MDA  before  advancing  to  the  next  phase  is  a 
form  of  governance  imposed  on  this  framework  Hence,  the  adherence  to  this  framework 
provides  the  PM  with  some  level  of  confidence  for  the  development  of  his  or  her  system 
It  is  also  not  a  rigid  framework  as  it  allows  the  PM  and  the  MDA  to  exercise  "discretion 
and  prudent  business  judgment"''*  to  tailor  the  number  of  phases  and  decision  points  for 
the  specific  needs  of  their  program. 

Due  to  the  complexify  involved  in  the  development  of  large*scale  systems,  it  is 
common  for  the  program  to  be  divided  into  smaller  projects  that  is  managed  by  the  PM. 
To  ensure  success,  it  is  imperative  that  the  PM  and  his  team  members  design  the 
framework  to  reflect  the  demands  of  the  program  and  define  the  required  inputs,  the  goals 
for  each  pha.se  of  the  process,  and  the  required  outputs  to  ramp  up  the  next  pha.se. 

C.  PROC  ESS  M  ODE  LI  NG 

Process  modeling  is  a  key  concept  in  process  engineering  The  model  forms  the 
basis  for  the  actual  process  for  use  in  the  development  of  a  system.  It  is  a  model  to 
describe  the  actions  of  processes  of  the  same  nature.  However,  the  model  is  just  an 
abstraction  that  requires  more  detail  to  be  filled  in  to  derive  the  actual  process.  It  is  not  a 
precise  representation  of  a  process  on  what  actually  happened  but  rather  a  rough 
anticipation  of  how  the  process  should  behave.  Hence,  the  model  has  to  be  refined  for  it 
to  work.  The  goals  of  a  process  model  are  as  follows:*^ 


DoD,  InsirucOon  5000  2 

'  ^  Wibpedia,  'Process  Modeling,”  hnpVi'en  wikipedia.or^Wilij/Proce8s_fncdeling,  (accessed 
December  8.  2009) 
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].  Descriptive 

a  Track  what  actually  happens  during  a  process 

b.  Takes  the  point  of  view  of  an  cxiemal  observer  who  looks  ai  the 
way  a  process  has  been  performed  and  determines  the 
improvements  ihat  have  to  be  made  lo  make  ii  pertbrm  more 
effectively  or  efficiently. 

2.  Prescriptive 

a  Defines  the  desired  processes  and  how  they  should/could/might  be 
performed. 

b.  Lays  down  rules,  guidelines,  and  behavior  patterns,  which,  if 
followed,  would  lead  to  the  desired  process  performance.  They  can 
range  from  strict  enforcement  ro  flexible  guidance. 

3.  Explanatory 

a  Provides  explanations  about  the  rationale  of  processes 

b.  Explores  and  evaluates  the  several  possible  courses  of  action  based 
on  rational  arguments. 

c.  E,st^lishcs  an  explicit  link  between  processes  and  the 
requirements  that  the  model  needs  to  fulfill. 

d.  Prc'dcfines  points  at  which  data  can  be  extracted  for  reporting 
purposes. 

Much  work  has  been  conducted  in  the  field  of  process  modeling.  There  exists 
workflow  technology  to  capture  business  processes  as  workflow  specifications  and 
increased  workflow  automation  in  complex  real* world  environmenus  involving 
heterogeneous,  autonomous,  and  distributed  information  systems.’^  In  conjunction  with 
this.  Business  Process  Modeling  (BPM)  is  used  to  represent  processes  in  an  enterprise.  It 
is  used  to  analyze  current  processes  in  order  to  improve  business  efficiency  and  quality. 
The  advent  of  visual  modeling  language  to  represent  business  processes  has  proven 
effective  for  presenting  it  to  business  stakeholders,  including  business  analysts  and 


Cl.  Diimiciias,  H.  M£rk,  aiwl  S.  AmJi,  ~An  oveniew  of  workflow  maiugemenl  Pn>m  prccesa 
modeling  to  workflow  aulomaOon  infrasinjcture,"  Drzinhitud  anti  Pai’o/lel  Da/ahasef  no  1  (April 
\99S}  II9-I53. 
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^srem  developers.  Supporting  technologies  include  Bu.<iness  Process  Modeling  Nolaiion 
(BPMN),  Unified  Modeling  Language  (UML),  modcUdriven  archiicclurc,  and  service* 
oriented  architecture  (So A) 

There  arc  also  formal  languages  to  provide  a  means  of  specifying  the  security 
properties  of  complex  systems  and  protocols.  Communicating  Sequential  Protocol  (CSP) 
is  such  a  language  for  describing  process  interaction  in  concurrent  systems.  It  is 
generally  applied  in  industry  as  a  tool  for  specifying  and  verifying  the  concurrent  aspecLs 
of  a  variety  of  different  systems  and  has  seen  active  work  to  increase  its  range  of  practical 
applicability,  for  exan^le,  increasing  the  scale  of  the  systems  that  can  be  tractably 
analyzed.*^  For  the  application  of  security  properties  software  and  the  analysis  of  those 
properties  In  the  context  of  a  well ‘defined  programming  environment,  there  are  formal 
approaches  developed  to  find  bugs  and  verify  the  absence  of  bugs  In  software.^ 

In  the  area  of  software  tools,  there  are  numerous  developments  to  automate 
solutions  to  support  process  design  and  operation  The  Alloy  Analyzer,  developed  by 
researchers  at  the  Massachusetts  Institute  of  Technology  (MIT),  is  one  such  tool  that  can 
be  used  to  analyze  ^eclfications  written  In  the  Alloy  specification  language.- ‘  The 
Analyzer  can  generate  Instances  of  model  invariants,  simulate  the  execution  of  operations 
defined  as  part  of  the  model,  and  check  user*specified  pr^erlies  of  a  model  As  the  tool 
supports  incremental  analysis  of  the  models.  It  is  capable  of  generating  immediate 
feedback  to  users  as  they  are  being  constmeted-a  term  known  as  analysis  of  partial 
models.^^  As  a  result.  It  can  perform  Incremental  analysis  of  models  as  they  are 
constructed  and  provide  immediate  feedback  to  users.  In  FUN  SOFT  nets,  an  approach  for 

' '  Wibpedia,  -Pto€«s  Modeling." 

A  W.  RoscDC,  7^  Theory  a/fd  Pf'acrice  of  Coftcufrency{Cm9i  Gniajn:  PrenliceHall  Europe, 

1W7). 

S  Creese,  "Data  Independent  Induciion:  CS?  Modd  Clieclung  of  Arbitrary  Sued  >^elwork»,~ 

(D  Phil  thesis,  Oxford  Univemry,  2001) 

^  H.  Chen  and  D  Wagner,  ~MOPS  An  mfcafrruclure  tdresaminmg  secunly  properties  of  software,'* 
CCS  “02'  Prceeediftgi  ihe  9th  ACM  Conference  on  Compuier  and  Commumcauons  Secunry,  (2002): 
235-244. 

^ '  D.  Jackaon,  Sc^rware  Absiraaiotii'  Lo^c.  Lang,ua^.  andAnaJytU  (MIT  Press,  2006). 

^  Wikipedia,  'Alloy  Arsalyze*,**  http  yyen.ttikipedia.orgA«ild/Alloy_Analyzer,  (accessed  December  ft, 
2009). 


modeling  and  analyzing  software  processes  was  innoduccd  (I  was  based  on  Petn  nets 
that  were  dcvelt^ed  to  support  software  process  modeling.  This  was  supposed  to  address 
the  shortcomings  of  Petn  nets  that  failed  (o  "measure  up  (o  more  detailed  requirements 
such  as  tight  but  still  comprehensible  representations  of  software  process  models.”^^ 

In  another  modeling  approach,  the  combinanon  of  UML  and  embedded 
Statecharts  was  applied  co  the  Unified  Cross  E>omain  Management  Office's  (UCDMO) 
cros'domain  solunons  workflow  process.-^  The  technique  of  using  Statecharts  for  the 
specification  and  development  of  a  complex  reactive  system  was  used  to  formally  specify 
and  reason  about  the  workflow  in  the  UCDMO.  In  this  approach,  the  StatcRover^^  was 
used  as  a  modeling  tool  for  the  workflow  processes  and  the  concept  of  embedded 
assertions  applied  to  the  workflow  to  check  that  specific  requirements  of  the  process  were 
adhered  to.  With  this  development,  the  process  engineer  can  formally  reason  the 
processes  and  conduct  inspection  and  analysis  of  processes  in  a  largely  human 'based 
workflow  of  the  UCDMO 

For  this  thesis,  the  author  models  the  workflow  of  the  eoncepUrefinement  phase 
using  an  UML  activity  diagram  The  author  also  utilizes  UML'like  Statechart  assertions 
to  specify  the  proper  and  timely  execution  of  the  tasks  layout  In  the  workflow  model.  The 
Statechart  assertions  are  developed  using  StateRover  for  the  Eclipse  integrated 
development  environment  (IDE),  which  supports  the  creation  and  validation  of  the 
Statechart  assertions  via  simulated  test  scenarios  within  the  JUnit  test  framework 


^  W  Emmench  and  V.  Cnihn,  "FUNSOPT  nets  a  ?etn>ne*  based  software  proce^  modeling 
language,"  ^ecificarion  and  Devgn.  h'oceedtng^  of  Sixth  Iniemauonal  Workshop  { IMI) 

l?5-(lt4. 
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M.  A  Schumann,  "A  Siaiechan  Model  of  the  Cross  Domain  ImplemenUtiun  Process,"  IATaC 
Nemiener  12  (February  2009)  26-30 

^  D.  Drusmsfcy,  hlodeting  and  Venjicauon  Unin^  UkfLSiaiechaii\  A  Working  Guide  to  Reactive 
Symem  Design,  Runtitne  Moniioring  and  Execuiion~hased  Model  Checking  (Burlington.  M A:  Elsevier, 
2006). 
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III.  THE  GQM  METHOD  FOR  CONCEPT  REHNEMEN  I 


A.  IMRODUCTION 

The  intent  of  this  chapter  is  to  introduce  the  GQM  approach  a.s  a  method  to 
develop  measurable  requirements  of  the  project  in  the  concept^refinement  pha.*ic.  The 
author  shall  illustrate  the  GQM  method  with  a  ease  study  involving  the  acquisition  of  a 
low'cost,  high*mobility.  advanced  low*altitude  missile  capability  Please  note  that  while 
the  ease  study  involves  an  aerual  air  defense  system,  the  speeificalions,  performances, 
and  so  on  are  not  specific  lo  the  system  per  se  It  is  used  solely  to  provide  a  more 
meaningful  illustration  of  the  application  of  the  GQM  model 

B.  DESCRIPTION  OF  THE  METHOD 

GQM  IS  a  lop'down,  goaUdriven  approach,  which  ensures  that  all  metrics  arc 
selected  for  a  goaUdnven  purpose  It  is  particularly  useful,  for  exan^le,  in  requirements 
definition,  because  every  requirement  should  be  measurable.  In  the  event  that  it  cannot  be 
measured,  either  the  user  has  to  review  and  redetine  it  or  that  particular  requirement  will 
not  be  included  in  the  specification  document  This  is  pertinent  as  it  safeguards  both  the 
interest  of  the  user  and  developer  when  the  system  is  finally  fielded  and  all  requirements 
are  measured  to  meet  the  specification.  In  essence,  what  cannot  be  measured  should  not 
be  considered  a  requirement 

The  GQM  method  was  developed  as  a  measurement  mechanism  for  feedback  and 
evaluation.  It  supports  program  planning,  for  example,  determining  the  cost  of  a  new 
program,  evaluating  the  strengths  and  weaknesses  of  the  current  processes  and  products, 
adopting/refining  techniques  as  well  as  evaluating  the  quality  of  specific  processes  and 
products  Measurement  also  helps,  during  the  course  of  a  program,  to  assess  its  progress, 
take  corrective  action  based  on  this  assessment,  and  evaluate  the  impact  of  such  action 


^  V  R.  Baeili,  "Data  Collechon,  Valtdaiion,  and  Analysis,'*  Tulon2}  on  \4odeli  and  Metrics /or 
Sofn^/are  Manageweftt  and  Bngtneeruig,  IEEE  Caulu^  no  EHO*  1 67.7  ( I  ^  I ):  31 0-3 1 3 
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(o  Older  io  improve  a  process,  the  organization  must  align  their  nicasuremcnt  goals  to 
corporate  goals.  These  goaK  can  then  be  franslaled  to  activities  that  can  be  definitively 
mcasuicd 

As  shown  in  Figure  2.  there  are  four  phases  to  the  GQM  method:^^ 

1  The  planning  phase  ts  performed  to  fulfill  all  basic  requirements  to  make 
the  GQM  nica.surcmcnt  program  a  success  This  includes  training, 
management  involvement  and  project  planning,  resulting  in  a  project  plan 

2  In  the  definition  phase,  all  deliverables  arc  developed  (goals,  questions, 
metrics  and  hypotheses  are  defined)  with  emphasis  on  structured 
interviews  and  knowledge  acquisition  techniques.  Actual  measurcmcois 
can  commence  when  the  definition  activities  are  convicted. 

3  In  the  data  collection  phase,  all  the  collected  data  arc  properly  stored  in  a 
measurement  database. 

4  In  the  interpretation  phase,  the  measurement  database  is  utilized  to  answer 
the  stated  questions  These  answers  arc  used  ogam  to  determine  if  the 
stated  goals  can  be  achieved. 


Figure  2.  The  Four  Phases  of  GQM 


Figure  3  shows  the  hierarchical  structure  of  a  GQM  model,  which  has  the 
following  three  levels:-^ 

1  Conceptual  level  (goal)  •  A  goal  is  defined  for  an  object  for  a  variety  of 
reasons,  with  respect  to  various  models  of  quality,  from  various  points  of 
view  and  relative  co  a  particular  environment, 


27  itjju  Soliogra  and  Egon  fieeghoul.  The  Cool'Qtuxrion'Metnc  Method  A  Practical  Cuide  for 
QuaJriy  fmprovemM  of  (Mc^w-Hill,  1999),  22-23. 

^Afbid. 
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2.  Operational  level  (question)  •  A  set  of  questions  is  used  to  define  models 
of  the  object  of  study  and  then  focuses  on  that  object  to  characterize  the 
assessment  or  achievement  of  a  specific  goal. 

3.  Quantitative  level  (metric)  •  A  set  of  metrics,  based  on  the  models^  is 
associated  with  every  question  in  order  to  answer  it  in  a  measurable  way. 

It  IS  intuihve  in  the  sense  that  all  questions  should  begin  with  a  goal  in  mind. 
Several  questions  can  be  raised  whereby  metrics  are  developed  for  each  question  to 
measure  the  outcome  Several  goals  can  also  have  questions  and  metrics  in  common, 
which  ensures  that  when  the  measure  is  actually  taken,  the  different  viewpoints  are  taken 
into  account  correctly;  that  is,  the  metric  might  have  different  interpretations  when  taken 
Irom  different  viewpoints.^  GQM  structures  of  goals,  questions  and  metrics  should  be 
built  upon  the  knowledge  of  the  experts  in  the  organization.  Particularly  in  the  definition 
phase,  where  knowledge  acquisition  techniques  are  also  applied  to  capture  the  implicit 
models  of  the  developers  built  during  the  years  of  experience.  Those  implicit  models  give 
valuable  input  into  the  measurement  program  and  will  ohen  be  more  important  than  the 
available  explicit  process  models.^® 


Figure  3.  Hierarchical  Structure  of  a  GQM  Model 


Basih,  "Data  Collection  " 

Sohngen  and  Eer^houi,  Gcol/Quesiron/XfeirK  Method,  22-23 
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Another  area  to  highlight  is  the  evolution  of  GQM  to  include  models  of  software 
processes  and  products.  The  result  is  a  modcl*bascd  GQM  approach^'  ihot  defines 
metrics  into  hvo  perspectives,  namely,  metrics  definition  by  members  of  the  project  team 
using  GQM  method  and  metrics  defmition  based  on  models  of  software  processes  and 
products,  as  shown  in  Figure  4.  The  intent  is  to  crosscheck  both  metrics  for  consistency 
and  con^lctencss  Once  this  is  completed,  the  actual  GQM  plan  is  dcveli^cd  The  GQM 
plan  consists  of  the  steps  in  a  measurement  program  to  document  the  goals,  related 
questions,  and  identified  metrics.  Measurement  can  start  once  the  plan  is  approved.  Data 
arc  collected  on  the  development  process  and  products,  aggregated,  and  validated. 
Finally,  the  mco^urcmcnl  results  are  retumed  to  members  for  analysis,  inlerprelaiion.  and 
evaluation  based  on  the  GQM  plan.^- 


PROJECT  TEAM 


SOFTWARE  PROCESSES  AND  PRODUCTS 


Figure  4. 


Metrics  Modeling  from  two  Perspw  lives 
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Soliogn  et  bI,  'AppheaDoo  ofSoftwan  Mrasurcmeal  el  Schlojuber^er  RFS:  Towards  Enhancm^ 
OQM,"  of  the  6“  European  iu/hvare  Control  and  Metrics  (ESCOM)  Confi}'ence.  May  l?-H, 

IW5 
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c. 


THE  CLAW  AIR  DEFENSE  SYSTEM” 


The  Complementary  Low*Aliitijde  Weapon  System  (CLAWS)  Is  operated  by  the 
United  Stales  Marine  Corps  (USMC)  to  achieve  combat  effeellveness  over  large  sectors 
of  the  batllespace.  The  system,  which  is  to  be  hlghly*mobilc  and  can  be  deployed  rapidly. 
Is  expected  to  possess  high  firepower,  be  all*wcaiher  capable,  and  be  equipped  with 
srandofT  capabilities  to  effectively  engage  current  and  emerging  air  threats.  The  types  of 
threats  Include  cruise  missiles  (CM),  unmanned  aerial  vehicles  (UAV),  and  advanced 
tixed*wing/rotary»wlng  (FW/RW)  aircraft. 

The  system  consists  of  the  Ml  097  Hlgh*Mobillty  Multipurpose  Wheeled  Vehicle 
(HMMWV)  with  a  minimum  of  four  Advanced  Medium-Range  Air*lo-Air  MLssile 
(AMRAAM)  missiles  mounted  on  a  launcher.  The  CLAW,  with  Ils  extended  range, 
lethality  and  accuracy,  Is  meant  to  be  operated  by  a  crew  of  two  The  concept  of 
operation  Is  to  complement  the  current  Low-Altitude  Air  Defense  (LAAD)  battalion  that 
operates  the  Stinger  system.  This  added  capability  will  provide  enhanced  protection, 
beyond  the  range  and  capability  of  existing  systems  In  the  LAAD  battalion.  The  new 
capability  will  also  maintain  the  hlgh*mobllity  required  for  organic  protection  of 
maneuver  elements  in  the  USMC 

One  of  the  main  dcv'elopmental  considerations  for  the  CLAW  program  Is  to 
maximize  the  use  of  current  Department  of  Defense  (DoD)  military  equipment.  This  will 
ensure  the  program  can  be  managed  In  a  cost*effectlve  manner  and  meet  short  delivery 
bmelincs  As  such.  It  Is  expected  that  the  program  will  take  liill  advantage  of  government 
off-the-shelf  (GOTS)  products,  commercial  off-the^helf  (COTS)  producLs,  government 
liimished  equipment  (GFE),  and  other  types  of  non-dcvelopmental  items  (NDI)  to  meet 
this  stringent  requirement.  Evident  from  this  requirement  Is  the  adaptation  of  the 
AMRAAM  system,  which  was  originally  developed  as  an  air-to-air  missile.  Likewise,  the 
platform  in  consideration  to  house  the  entire  system  is  the  HMMWV  vehicle,  which  Is 
used  by  the  Avenger  Air  Defense  System,  also  operated  by  the  LAAD  battalion. 


”  GlobaJSecutiry.oc^  hnp'^/www.globttlseciuity  0f^/i7iil)i£ryi«y«lefnS''miiniiinnvcli(ws  hon  {accessed 
November  1 5.  2009) 
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Figure  5.  Tbc  CLAWS  (HUMRAAM)  Surface-launched  AMRAAM  Air  Defease 

Missile^ 

Besides  short  dcsclupmcnl  lime  and  loxv-nsk  concerns  (especially  budgetary 
concerns),  one  of  the  key  advantages  for  such  development  consideration  is  that  the 
systems  are  all  fielded  systems  and  combat-proven  While  tt  is  based  on  mature 
technology,  the  capability  is  sufficient  to  meet  the  specified  threats.  Additionally,  the 
developed  system  can  easily  integrate  \S'ith  current  Command,  Control.  Communicadon 
and  Intelligence  (C31)  architecture  and  ioletface  with  the  suite  of  available  sensors  for 
full  air-situation  awareness.  In  short,  it  is  a  system  that  is  ready  to  be  deployed  for  action 
once  It  is  delivered  without  concerns  of  the  many  inidal  problems  that  often  beset 
systems  based  on  new  technologies.  The  Operations  and  Support  (O&S)  phosc  of  the 
program  ts  also  expected  to  be  more  efficient  and  supportable  due  to  the  economics  of 
scale  in  sharing  similar  components  and  expertise. 

D.  APPLICATION  OF  THE  GQM  METHOD  TO  THE  ACQL’ISITION 

SYSTEM 

This  section  presents  an  application  of  the  GQM  method  to  the  development  of 
the  CLAW  system.  The  method  is  useful  in  that  it  helps  to  provide  better  requirements 
analysis.  ReferriDg  back  to  the  acquisition  cycle  in  Chapter  II,  the  output  of  this  analysis 
will  serve  as  part  of  the  formulation  of  the  TDS  document,  which  then  serves  as  the  input 


Net  Resource  IntemalionaJ,  '^urttce -Launched  A.\dtlAAM  (SL-AMRAAM  /  CLAWS  I  Medium- 
Rooge  Air  Defense  SysteiD,  USA,"  hllpi.'.Wxra  anny-Kchnology  coiWpioiecii.'turrace-launchedsurrsce- 
launched  I  .bOnJ  ( occeiscd  November  I S,  2004) 
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10  the  technology  dcvcl^mcnl  phase.  Specifically,  the  stated  requirement  which  is 
extracted  from  the  ICD  is  ro  develop  a  low*cost,  high*mobility,  advanced  low*allirude 
mi&sile  capability.  The  intent  is  lo  develop  strategics  and  measurable  metrics  that  will 
answer  directly  to  the  staled  requirement  Measurements  also  help  during  the  course  of  a 
pro}ecl  to  assess  its  progress,  take  corrective  action  based  on  this  assessment,  and 
evaluate  the  impact  of  such  action.  As  stated  in  an  earlier  chapter,  the  figures  used  are 
fictitious  and  arc  used  solely  to  provide  a  more  meaningful  illustration  of  the  application 
of  the  GQM  methodology 

The  GQM  method,  as  recommended  by  Basil!  and  Rombach,^^  begins  with  a 
tmmework  for  the  construction  of  goals.  In  the  design  of  an  actual  acquisition  system, 
there  would  be  many  strategics  for  such  a  requirement  However,  as  an  illustration  lo 
demonstrate  this  approach,  only  three  strategies  are  established  as  shown  in  Figure  6  As 
described  and  shown  In  the  figure,  the  CLAW  requirement  was  lo  develop  a  low*cost, 
high*mobilify,  advanced  low*allitude  missile  capability.  In  order  to  satisfy  this 
requirement,  three  strategies  were  developed  as  follows. 

51  *  To  maximize  the  use  of  existing  technologies  and  DoD  military  equipment. 

This  will  ensure  the  program  can  be  managed  in  a  cosl*effeclivc  manner  and  meet 

short  delivery  timelines 

52  :  To  meet  the  mobility  requirement  for  the  system  to  keep  pace  with 

expeditionary  and  supported  maneuver  elements. 

53  :  To  maximize  the  use  of  NDI  including  an  unmodified  AMRAAM  missile 

With  these  three  snategles.  goals  were  then  developed  for  each  of  the  strategies. 
The  number  of  goals  varied  depending  on  the  complexity  of  the  strategies.  In  this 
example,  four  goals  were  set  to  meet  the  strategies  for  the  requirement.  For  the  first 
strategy  (SI),  the  goal  (Gl)  was  to  adapt  from  the  existing  LLAD  support  system 
However,  it  is  not  possible  that  all  of  the  existing  LLAD  support  system  is  compatible  lo 
support  the  newly  developed  system.  Hence,  the  question  (Ql)  was  raised  with  regard  lo 
what  specific  types  of  support^maintenance  systems  were  available  lo  meet  this  goal.  As  a 


V  R  Basili  «Jid  H  D.  Rombach,  "The  TAME  Ptqeci  Towards  Impravemeni-Onented  Soflwtire 
EnvironmenB,"  IEEE  Tra/vactK’tn  on  E»^»eerin^  SE-14,  i»o.  6  (June  19S8):  761 
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means  to  ensure  that  the  strategy  was  sufficiently  accomplished,  the  metric  (Ml)  for  this 
goal  was  the  extent  (in  terms  of  percentile)  that  current  DoD  inventory  was  used. 
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Figure  6. 


The  CLAWS  GQM  Model 
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For  (he  second  stra(egy  (S2),  the  goal  (G2)  was  to  use  an  unmodified  Avenger 
HMM  WV.  For  the  vehicle  to  be  suited  for  the  newly  developed  sysiem.  the  question  (Q2) 
that  was  raised  dealt  with  ensuring  that  the  HMMWV  Is  completely  sclf*sustalned  such 
that  It  Is  capable  of  carrying  all  equipment/peisonnel  needed  for  its  mission.  Tbe  metric 
(M2)  for  this  goal  was  the  extent  (also  in  terms  of  percentile)  (hat  the  equipment  can  be 
contained  In  the  vehicle  Another  goal  (02)  was  set  to  hirther  support  this  strategy.  The 
goal  was  to  meet  mobility  needs,  and  in  this  case,  be  air*Qanspor1able  to  meet  mission 
requirements.  The  question  (Q3)  (hat  was  raised  was  concerning  the  accommodation  of 
the  vehicles  Into  a  C*130  transport  plane,  with  the  metric  (M3)  for  this  goal  being  the 
number  of  vehicles  that  was  required  to  fit  into  (he  C*  1 30  transport  plane. 

For  the  last  strategy  (S3),  a  decision  was  made  earlier  to  adopt  a  NDl  approach, 
and  hence  the  AMRAAM  missile  was  chosen  as  the  goal  (G4)  to  fultlll  this  strategy. 
However,  as  the  AMRAAM  missile  was  originally  developed  for  alr*to«alr  combat 
missions,  (here  was  a  need  to  ascertain  (hat  It  can  be  successfully  modified  for  land* 
based,  air  defense  mission.  The  three  questions  (Q4,  QS  and  Q6)  (hat  were  asked  were 
made  ro  ensure  that  the  AMRAAM  missile  for  the  CLAW  system  can  be  modified 
successfully  to  complement  (he  LLAD  force  structure  and  Integrated  seamlessly  to  (he 
existing  C3(  infrastructure.  Consequently,  three  metrics  (M4,  M5  and  M6)  were  used  as 
unit  of  measurement  for  the  required  performance  parameters  for  this  goal. 

The  metrics  only  provide  the  unit  of  measurement  for  each  stated  goal.  To  ensure 
the  requirements  were  quantifiable,  they  were  further  refined  Into  derived  requirements 
(DR)  based  on  the  metrics  The  respective  DRs  for  the  metrics  are  shown  In  Figure  6. 

As  can  be  seen  In  (he  figure,  contextual  information  was  provided  where 
necessary  so  that  project  team  members  could  relate  to  the  intent  of  the  requirement  for 
them  to  exercise  better  analysis  of  the  program.  In  this  case,  (he  context  for  this 
requirement  was  for  the  developed  system  to  be  compatible  with  the  existing  force 
structure  and  C31  infrastructure.  With  this  context,  (he  project  team  would  then  be  able  to 
scope  their  development  strategies,  taking  into  consideration  interoperability  and 
integration  issues. 
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For  instance,  to  illustrate  the  role  played  by  contextual  information,  it  was  written 
in  earlier  paragraph  that  there  was  one  goal  (Gl)  to  meet  strategy  SI.  and  it  was  to  adapt 
from  existing  LAAD  support  systems.  Relating  from  the  earlier  context  of  being 
compatible  with  the  existing  force  stnieture.  this  goal  was  chosen  not  only  to  meet 
strategy  SI.  which  was  to  maximize  the  use  of  existing  technologies  and  equipment  so  as 
to  minimize  cost  and  the  delivery  schedule,  but  also  to  ensure  compatibility  in  terms  of 
maintenance,  training,  spares  support,  etc.  to  the  existing  systems  in  the  LAAD  combat 
and  service  support  structure.  Achieving  this  would  further  ensure  reduced  O&S  costs  to 
the  life  cycle  of  the  system.  This  exemplified  the  need  for  the  project  team  to  know  the 
context  of  the  program  beyond  the  capabilities  and  performance  parameters  of  the  combat 
system  In  another  example,  the  goal  (G3)  for  strategy  S2  was  to  be  C*  130  transportable. 
The  question  raised  was  to  ascertain  the  number  of  vehicles  to  be  transported.  In  this 
case,  the  derived  requirement  (DR3)  indicated  three  vehicles  were  sufficient  It  might  be 
trivial  as  this  metric  could  be  easily  obtained  by  checking  with  the  operators  of  the 
system  However,  the  context  given  to  the  project  team  was  that  a  LAAD  fighting  unit 
comprised  three  fire*umts.  Hence,  the  C*130  must  be  able  to  accommodate  this  number. 
However,  beyond  that  with  this  context  the  project  team  could  also  bear  in  mind  this 
configuration  in  other  analysis.  In  essence,  their  development  should  be  done  with  the 
correct  context  in  mind  and  not  in  isolation. 

In  summary,  the  GQM  method  supports  the  process  to  work  systematically 
downwards  where  questions  can  be  raised  to  achieve  more  clanty.  Together,  the 
questions  and  metrics  arc  identified  to  fulfill  the  goal  The  objective  is  to  produce 
mea.surable  metrics.  These  monies  can  then  be  further  refined  with  the  stakeholders  (for 
example,  system  aerators)  and  established  as  a  form  of  a  measure  of  performance  during 
the  OT&E  as  part  system  and  development  phase  of  the  program 
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IV.  ACQtlSniON  PROCESS  MODELING  AND  ASSURANCE 


A.  INTRODUCTION 

In  (his  chapter,  (he  UML  am^cts  and  the  Statechait  asaer1ion.s  were  used  as  light* 
weight  fonnal  approaches  to  model  and  verify  the  correct  behavior  of  (he  system.  To 
achieve  this,  (he  UML  activity  diagram  was  used  as  a  scmi* formal  approach  to  model  (he 
workflow  process  of  (he  conccpt*rctinement  pha.se  m  an  acquisition  process  It  is  a 
suitable  analysis  tool  for  this  purpose  as  it  can  be  used  to  describe  the  worktlow  in 
varying  levels  of  detail  These  details  can  then  be  u.scd  to  construct  Statechart  assertions 
using  (he  StateRover  tooP^  to  specify  fonnal  assertions  and  to  ensure  the  proper  and 
timely  execution  of  the  acquisition  (asks  layouts  in  the  acquisition  process  model 

B.  ACQUISITION  PROCESS  MODELING 

I.  Use  Case  Aoatysik 

A  use  case  in  software  engineering  and  systems  engineering  is  a  description  of  a 
system's  behavior  as  it  responds  to  an  event  (hat  originates  from  outside  of  that  system. 
In  other  words,  it  is  a  scenario  to  illustrate  some  sequence  of  interactions  between  a  user 
and  a  system  so  as  to  capture  the  system's  behavioral  requirements  by  detailing  scenario* 
driven  processes  through  the  functional  requirements.^^  “The  use  cases  help  the  modelers 
understand  the  problems  to  be  solved  and  the  ob)eetives  to  be  accomplished  by  (he 
perceived  system.  The  high*level  use  eases  are  goal^iriented,  and  typically  arc  used  to 
describe  the  workflow  of  a  business  process  instead  of  interactions  among  system 
components.  Mapping  the  scenarios  of  the  use  cases  to  activity  diagrams  helps  highlight 
the  assignment  of  responsibilities  and  the  interdependencies  among  the  different 
components  (of  an  organization  or  system)”.^^ 

Simon  Bennen,  John  Skelton  and  Ken  Lumu  Schoum 't  Owlines  of  UML  (McGtbw-HiII 
IntemalionsI,  2001),  208 

Dnisinsky,  Modeling  and  Venficaiion 

Use  Ca.<ie,  hnp:''en  wilijpedja.or^'ttiki/U8e_case,  (assessed  December  8,  2009) 

D  Dnjsinidty,  J  B  Michael  and  M.  Slung,  ~A  Fiamewock  (bi  Compuler^Aided  ValidaOon,'* 
hno'^uon^  rn  and  Software  Enginee>  ing,  4(2),  June  2008, 161-168. 
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To  undcniland  the  requirements  and  constraints  for  the  acquisition  process,  three 
use  cases  were  developed  to  describe  the  tasks  in  the  conccpt*rcfinemcnl  phase  for  the 
development  of  the  TDS  document  Specifically,  the  use  cases  describe  the  process  on 
the  interaction  between  the  concept-refinement  team,  PM,  system  operators  and  approval 
authority  The  role  of  the  concept*refinemenl  team  is  to  lake  the  requirement  from  the 
(CD  and  performs  analysis  to  break  down  the  high  level  requirements  into  measurable 
goals.  To  do  this,  (he  concept*refinement  team  must  analyze  different  altemahves  and 
adopt  appr^riate  strategies  to  achieve  (his  requirement.  Different  sets  of  goals  are 
developed  to  meet  these  strategies  and  for  each  of  these  goals,  measurable  metrics  must 
be  defined.  The  conccpt-rcfincmcnt  team  then  develops  and  submits  the  draft  TDS 
document  to  the  PM.  Upon  the  receipt  of  the  draft  TDS  document,  the  PM  will  arrange 
and  schedule  for  a  review  with  the  system  operators.  If  the  system  operator  is  satisfied 
with  this  review,  the  PM  can  forward  the  draft  TDS  document  to  the  MDA  for  approval. 
However,  should  any  of  the  analysis  falls  short  of  the  requirement,  the  PM  will  task  (he 
concept-refinement  team  to  further  revise  the  TDS  document  and  adjust  the  project 
schedule  accordingly.  When  the  TDS  document  is  finally  submitted  to  the  MDA.  the 
MDA  IS  to  review  and  approve  (he  details  of  the  analysis  If  the  document  is  approved, 
the  concept-refinement  phase  is  considered  complete  and  the  approved  TDS  document 
will  be  used  as  (he  input  for  the  next  phase  of  the  acquisition  process  Otherwise,  (he  PM 
and  his  conccpt-rcfinemcnt  team  are  required  to  conduct  more  analysis  based  upon  the 
direction  given  by  the  MDA.  (n  addition,  the  PM  has  to  review  his  project  schedule 
again 
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Figure  7.  Use  Case  Model 

Figure  7  shows  the  model  of  the  U5e  cases  for  the  concept*rcfinemcnt  phase. 
Three  use  cases  that  were  used  as  an  analysis  are  as  follows: 


Use  Case:  UC>1  Requirements  Analy^b 

Primary  Actor  Conccpt*Rcfinement  Team  (CRT) 

Other  Actors:  PM 
Stakeholders  and  Interest: 

•  The  CRT  needs  to  consider  all  alternatives  and  develop  the  TDS 
document. 

•  The  PM  wants  to  complete  the  concept^refinement  phase  in  time  and  get 
the  TDS  document  to  be  approved  by  the  MDA. 


Entry  Conditions:  Receipt  of  all  the  input  documents. 

Success  Guarantee:  Produce  the  TDS  document 


Typical  Flow  of  EvenU: 

1 .  Extract  and  breakdown  requirements  from  the  ICD 

2.  For  each  requirement,  consider  all  the  allcmaUves  In  the  AoA 
plans. 

3.  Select  srraiegies  lo  achieve  the  requirement. 

4.  Select  goals  (o  meet  this  strategy. 

5.  Formulate  metrics  and  DRs  lo  measure  the  performance 

6.  Submit  the  draft  TDS  document  lo  the  PM  for  review. 

Alternate  Flows: 

]  a.  Incomplete  information  lo  perform  analysis. 

] .  Inform  PM  of  the  problem  and  reset  the  project  schedule. 

2.  Contact  the  acquisition  sponsor  agency  and  proceed  lo  Step 
2  upon  receipt  of  the  missing  information. 

Special  Requirements:  Thirty  days  timeline  ro  complete  the  concept* 

refinement  phase. 

Use  Case:  lfC-2  TDS  Review 

Primary  Actor  PM 

Other  Actors:  System  Operators,  CRT 
Stakeholders  and  Interest: 

•  The  PM  and  the  CRT  want  lo  review  the  completed  TDS  document  lo 
ensure  It  meets  the  requirements  of  the  system  operators. 

•  The  system  operators  want  lo  ensure  that  all  system  specifications  arc  met 
in  accordance  to  the  system  requirements. 

Entry  Conditions:  The  completed  TDS  document. 

Success  Guarantee:  The  TDS  document  is  reviewed  without  further  analysis 
required. 

Typical  Flow  of  Events: 

] .  PM  contacts  and  arranges  with  system  operators  lo  review  the  TDS 
document. 

2.  PM  forwards  the  TDS  document  lo  the  system  operators  ahead  of 
the  review 

3.  PM  conducts  review  (attended  by  the  CRT)  with  the  system 
operators. 
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4.  TDS  document  is  reviewed  without  tiiither  analysis  required  and  is 
accepted  by  the  system  operators. 

Alternate  Flows: 

4a.  System  operators  raised  issues  with  the  TDS  document. 

].  CRT  to  establish  a  plan  to  address  the  issues  and  submit 
revised  TDS  document  to  PM  for  review 

2.  PM  to  revise  schedule. 

Special  Requirements.  Thirty  days  timeline  to  complete  the  concept* 

refinement  phase. 

Use  Case:  llC-3  Approval  of  TDS 

Primary  Actor  PM 

Other  Actors:  MDA 
Stakeholders  and  Interest: 

•  The  MDA  wants  to  ensure  that  all  parameters  required  in  the  TDS 
document  are  included  and  the  review  with  the  system  operators  is 
conducted  without  any  turthcr  review  required. 

•  The  PM  wants  to  seek  approval  of  the  TDS  document  and  complete  the 
concept^refincment  phase  in  time. 

Entry  Conditions:  The  reviewed  TDS  document. 

Success  Guarantee:  The  TDS  document  is  approved  for  the  next  phase  of  the 

acquisition  process 
Typical  Flow  of  Events. 

1 .  PM  submits  TDS  document  to  MDA. 

2.  MDA  conducts  review  of  the  TDS  document 

3.  TDS  document  Is  approved. 

Alternate  Flows: 

3a.  TDS  document  Is  not  approved. 

] .  PM  has  to  create  a  new  task  and  revise  the  schedule. 

Special  Requirements.  Thirty  days  timeline  to  complete  the  concept* 

refinement  phase. 
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2. 


Wi»rkl1(m'  Modeling 


The  next  stop  is  to  model  the  workflow  In  terms  of  acnvjfy  diagrams  The 
flexibility  of  the  activity  diagram  allows  it  to  be  used  in  a  variety  of  ways.  It  can  be  used 
TO  deT^nbe  business  flows  in  varying  degrees,  complex  flows  within  or  between  use 
cases,  or  complex  behaviors  within  an  object.^^  For  the  author's  application,  the  activity 
diagram,  as  shown  in  Figure  8,  is  used  to  describe  the  flow  of  activities  between  diflerent 
actors  during  the  coneept*refinement  phase  of  the  acquisition  process. 

While  the  actual  concept*relincment  phase  does  exist  in  the  U.S  DoD  acquisition 
process,  the  activities  presented  in  Figure  8  are  pieced  together  based  on  other  research 
materials,  it  is  just  a  fictitious  representation  and  not  based  on  an  actual  project  by  the 
U.S.  DoD 

As  mentioned  before,  the  activity  diagram  resembles  the  workflow  process  of  the 
concept*rcfincment  phase  The  key  Input  to  kick  start  this  phase  is  the  receipt  of  the 
necessary  input  such  as  the  ICD  and  AoA  plans.  The  concept*rcfinemcnt  team  is  required 
to  use  these  documents  to  select  the  best  options  that  meet  the  requirements  to  develop 
the  TDS  document  for  the  next  phase  of  the  program.  To  do  this,  the  team  is  to  consider 
various  alternatives  for  each  requirement  and  consider  user  input^feedback  as  well.  The 
phase  IS  considered  convicted  when  the  developed  TDS  is  approved  by  the  MDA 

To  elaborate,  the  conccpt*rcfinement  phase  will  commence  upon  the  receipt  of  the 
necessary  input  documents,  notably  the  ICD  and  the  AoA  plan.  The  eoncepr-refinement 
team  is  required  to  consider  at  least  two  alternatives  as  per  the  AoA  plan  for  each  of  the 
requirements  that  they  have  selected.  Once  the  alternatives  have  been  considered  and  the 
options  weighed,  the  team  can  proceed  lo  select  a  strategy  for  developing  this 
requirement. 


^  D.  Druemsky,  I  B  Michael  and  M.  Shing,  ~A  Ftamewock  tot  Compuler-Aided  Validabon,' 
hno'^liomi  rn  and  Sofn^'ore  Engmeeung,  4(2),  June  200^,  161— 1 6S. 


Figure  8.  Activity  Diagram  for  Concept* Refinement  Phase 
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Following  the  GQM  model,  the  goals  for  the  selected  strategy  must  be  developed, 
and  questions  could  be  raised  as  a  result  of  these  goals  Finally,  the  metrics  and  the  DRs 
can  be  developed  to  fulfill  the  measurement  of  the  respective  goals  The  process  is 
iterative  and  considered  complete  when  all  requirements  arc  decomposed  with  the  GQM 
model.  Upon  completion,  the  concept^refinement  team  can  proceed  to  develop  the  draft 
TDS  document  and  the  PM  will  arrange  with  the  stakeholders  to  conduct  a  review. 
Depending  on  the  decisions  by  the  stakeholders  and  project  team,  it  is  possible  for  some 
requirements  to  be  further  analyzed  When  this  happens,  the  PM  has  to  revise  his  project 
schedule  Otherwise,  the  TDS  document  is  submitted  for  approval  by  the  Milestone 
Decision  Authority. 

C.  STAT  ECH  ART  ASSERTION  S  DEV  E  LOPM  ENT  AND  V  A  L I D  ATI  ON 

Harel  Siatecharts^'  are  commonly  used  in  the  design  analysis  phase  of  an  object* 
oriented  UML*bascd  design  methodology,  for  example,  Brugge  suggests  using 
Staiechaits  in  the  design  analysis  phase  of  an  objecl*oncntcd,  UML*based  design 
methodology  to  specify  dynamic  behavior  of  complex  reactive  syslems.^^  Stalechart 
assertion  is  a  formalism  that  combines  UML*based  prototyping,  UML*based  formal 
specifications,  runtime  monitoring,  and  execution*based  model  checking. The  main 
advantage  for  using  this  methodology  is  that,  unlike  temporal  logic 'based  specification 
languages,  which  are  purely  propositional,  Statechan  assertions  are  more  Intuitive  to  use 
because  they  are  visual  and  closer  to  the  thinking  of  the  system  designers  in  the  modeling 
of  the  system  behaviors. 
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D.  Hare],  ~A  Visual  Formalism  for  Complex  Systems,"  Sctence  of  Computer  8,  no.  3 


(1«7):  251-274 
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B  Bniepge.  Ohject'Oriented  Software  Engi/*eeri»^.  Ua iwg  VXtL,  Pattenvi,  and  Java,  2nd  ed. 


(PreniiM  Hall,  2004). 

D.  Drusmsky,  'Semajilics  and  Runtime  Morutoiin^  of  TLCharb  Statechan  Aulomala  wilh 
Temporal  Logic  CundiOuned  Transitions,"  Proc  4th  ku/tume  Venficarioit  Woththop  (kV  04/.  Electronic 
Notes  tn  Theoretical  Compute!-  Science  1 13,  (200S):  5-21 

Drusinsky,  Modeling  and  yenfication 
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The  author  developed  Stalechan  assertion.^  using  the  SlaicRovcr  iool^^,  which 
provides  support  for  design  entry,  code  generation,  and  visual  debugging  animation  for 
UML  Slaiecharts  combined  with  How  charts  to  provide  runtime  monitoring  and  runtime 
recovery  trom  assertion  failures. 


Typically,  formal  specifications  arc  created  from  a  conceptual  requirement  as 
understood  by  the  primary  modeler.  Regardless  of  the  formal  notation  or  method  used, 
system  modelers  typically  begin  their  requirements  discovery  process  using  some 
scenarios  that  Involve  the  system  and  Its  environment.  They  first  express  their 
understanding  of  the  expected  behavior  or  properties  of  the  system  informally  using 
natural  language  and  then  translate  the  assertion  into  formal  assertions  following  the 
process  shown  In  Figure  9  ** 
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Figure  9  Iterative  Process  for  Assertion  Development 
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Using  ihe  aellvily  diagram  in  Figure  8,  ihc  PM  first  identifies  the  events  that  are 
relevant  to  the  proper  and  timely  execution  of  the  tasks  in  the  conccpt*rctinemcnt  phase 
as  shown  in  Table  I . 


Events 

Representation 

1 .  The  receipt  of  the  necessary  input  documents 

Receive  input 

2.  The  analysis  of  the  alternative  plans  has  been  considered  for  each 

of  the  requirements  stated 

Ahernouve  and 

requirement 

selected 

3.  Strategies  for  each  requirement  arc  selected 

Strategies 

selected 

4.  The  goals  are  developed 

Goals  developed 

5.  Questions  as  a  result  of  the  goals  are  raised 

Questions  ratsed 

6.  The  metrics  are  developed 

Metrics 

developed 

7  The  review  by  the  stakeholders  arc  complete 

Review 

completed 

8.  The  TDS  has  been  completed 

TDS  completed 

Tabic  1.  Events  of  Interest  for  Statechart  Assertion  Model 

The  PM  then  proceeds  to  express,  in  natural  language,  the  scenarios  that  describe 
the  proper  and  timely  execution  of  the  tasks  based  on  the  observable  events,  as 
exemplified  by  the  following  two  assertions 

Assertion  I  :  The  concepl^refinemcnt  team  muxi  complete  the  Technology 
Development  Strategy  fTDS)  document  within  eight  weeks  after 
receiving  the  necessary  documents. 

Assertion  2  :  The  concepl^refinement  Team  must  contdder  at  least  mo 
alternatives  for  each  requirement. 

Next,  the  PM  translates  Assertion  I  into  the  Statechart  assertion  shown  in  Figure 
10  and  tests  the  correctness  of  the  assertion  using  the  two  test  cases  shown  in  Listings  I 
and  2. 
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Figure  10.  Siaiechan  for  A^rtion  I 


The  assertion  is  v^Tlrten  from  an  observer's  poiot  of  vicv/  wbo  wants  to  assure  that 
the  TDS  document  can  be  completed  wiihin  thirty  days  from  the  receipt  of  the  necessary 
input  documents.  The  Statcchart  assertion  is  uitercstcd  in  the  events:  rcceiveinput, 
idsCompleied  and  limeoutFire.  Upon  receipt  of  the  documents,  the  Statcchart  assertion 

*dl  enter  the  RejiningConcepts  state  A  lhuty*day  timer  will  then  be  fired  to  keep  track 
of  the  progress.  If  the  document  ts  completed  in  a  timely  fashion,  it  will  enter  the 
RcfinementCompiele  state.  Otherwise,  the  assertion  would  fall  and  the  timeoutFUe  event 
would  lead  co  an  Error  state.  To  assure  that  the  assemon  works  as  specified,  the  AJnit 
test  case  in  Listiogs  1  and  2  is  used  to  test  the  behavior. 
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Inpo  rt  j  uniC . f  raraewo  rk . Ta  st  Ca  s  0 ; 

piihlic  class  AAaertioal_IeBtl  axeanda  TesCCa^e  | 
privaca  Assaatianl  al  •  null: 

protected  void  aetUpO  tbrowa  1 | 

lupar.aetUp () ; 
al  ■  new  Asaartlosl { > : 

I 

protected  void  tear  Down  ()  throws  \ 

al  ■  null: 
super. tear Down  ( ) ; 

1 

public  void  teatAsaertiool 0  | 
a 1 . rece iva Input i ) ; 

this .aasectTrue  4  al. isState ("Ref inlnqConcept  s* ) \ ; 
al ,  lftccTiiive(29} : 

thia.assertTrueUl.leState  ("Ref  inlngConcepts" ; )  / 
al.tdaCoo^letedl \ ; 

this .aasectTrue (a 1. isS tat a ("Ref inaoeotCooplet a” } ) ; 
this .aasectTrue ( al . laSucceaa ( ) ) : 

I 

) 


Li&dng  1.  Test  Case  I  for  A:»8erlioD  1 


Tesi  rase  I  is  setup  to  highJight  that  positive  activity  has  taken  place  that 
complied  \mth  the  assertion  statement.  It  is  what  the  author  termed  as  a  “happy"  scenario. 
In  this  case,  the  TDS  document  is  completed  wiftun  thirty  da^s  and  the  Statechait 
assertion  enters  the  RefinenientCompleie  stale.  Test  rase  2  represents  an  exception  to  the 
rule,  where  the  time  limit  of  thirty  days  is  exceeded,  causing  the  Statcchart  assertion  to 
enter  the  Error  state,  and  signaling  a  violauon  10  the  asscraon, 
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iispost  jualt .  f raioevork  .TeatC&ae; 


public  claaa  AaBecClaQl_TeaC2  cxtenda  TeatC&ae  | 
pcivatc  ABasrtionl  al  •  oull; 

pcatscted  void  setUpO  tbcawa  EX'^npruri  ( 

Bupec.setTTp  I  ]  / 

aL  *  nev  Aaaectlonl f) ; 

) 

protactad  wold  taatDovmO  throvpa  ( 

aL  “  nullj 
Bupec.'tearDawn  |  f  / 

1 

public  veld  teatAasectloDl | f  | 
lot  i  ■  0; 
a  L . racel ve  t  nput { ) ; 

thia.aaaartTrue ;al . laScata ("AaflnlngConcapta”) ) ; 
al .  inctTiine  <  31 )  j 

tbla.asaercTrQB (al . laStaCe | "Error " ) ) ) 
a 1 . tdaCompl a  tad ( ) ; 

thlB-aBasrcTroB (al , laStats | "Error" ) ) j 
thlB .aBaBrtFalaa (al , iaSuccaaa | ] ) ; 

1 

} 


Li&dng  2.  TbsI  Case  2  for  Asserlioa  1 


Figure  1 1  shows  the  Stateclurl  assertion  for  Assemon  2.  that  ihc  concept* 
refinement  team  musi  coosidcr  at  least  two  altemaiivcs  for  each  requireineni 
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Figure  1 1 .  Staiccbon  for  Assertion  2 

The  corrcclJiess  of  the  Slaicchart  assertion  is  validated  with  three  JUnil  lest  casc5 
shown  in  Listings  3>  4  and  S.  Here,  the  data  stnichire  cni  is  used  to  keep  track  of  the 
number  of  alternatives  that  were  considered  with  three  sequencing  events: 
se/ecfRe^uiremeni,  selecfS/rate^\  and  comp/eieCQM.  For  every  requirement  selected, 
the  Stalechart  assertion  will  enter  the  Re<fSelecied  stale.  Within  this  state,  the  concept* 
refinement  team  will  be  monitored  so  that  they  consider  at  least  two  other  strategies. 
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loipart  junlC.  fraoiework.  TaacCaja; 

public  cLasj  ?LaaeEtioa3_rejtl  axtanda  TeaCCaae 

1 

pi^lic  void  teatAaaertiool  I)  ( 


1 

a2 ,  selact  Aaqulraoienc  {e}  i 

chLa  .aasacCTnie  |a2.  iaSCace  ("ReqSelcctad''  \ )  / 
a2 . aelactScracegy {al j ; 
thLa • asaaECTcua 4a2 .cnt  I); 

chLa  .aaaacCTnie  |a2 .  iaStace  ('’RaqSelactad") )  t 
a2.aelectBtracegv  (a2)  ,* 
chta>asBaECTcu«U2.cnt  ••2); 
chLa.a6sacCTnia|a2.iaStace  ("ReqSelactad'*]  )  t 
a2. cooplaceGCM  (c} ; 

thia ,  asaaEttcua  U2 .  iaScata  ; 

thla  >  asaaECTrua  4a2 . iaSuccaaa (j ) ; 

1 

Li&dng  3.  Tci»i  Case  1  for  AjiscrlioD  2 

loporC  juaic.  f raneaioEk.TeaCCaae ; 

public  data  AaaeECiaa2_TeBt2  axtanda  TeatCa^e 
public  void  teetAssertlonl 0  | 


1 

1 

a2 « aelaccRequlramant (e)  ; 

thia  .assaEtTEua  \  a2 .  laState  ('’AaqSalaccad"  * ) ; 

a2.aelectBtEatagy (al) / 

thla .aaaectTrua 4a2.cnt  1); 

thla  .aasaEtTEua  \  a2 .  laState  ('’AaqSalactad'*) ) ; 

a2.aelecCBtEatagy (a21 / 

thla.aaaectTEue4a2.cnt  ••  1); 

thia .assaEtTEue 4 a2 .laState  ("AaqSalactad'’ * ) ; 

a2 ,  cooiplateSOH  (E  ^ ; 

thia .aaaectTrua  4  a2. iaStata  CError “ \ ) / 
thla.aaaectfalae (a2 .IaSuccaaa  () f ; 

Listing  4.  Test  Case  2  for  Assertion  2 
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loipart  juaiC.  fracsework.TeatCaae; 


public  =Las9  ?UBertiaa2_7ejt3  sxtsnda  TeatCaae  ( 


public  void  teatXaBeEtiaal ( f  ( 
a2  ,  salectftaqu  ireoien  c  ( r  f  ,* 

tnii . aBBSEEtEue (a2 . iaScate  f "^aqdelBcted” ) ) i 

a2. BelectStrategy  (b1 ) ; 

tnii . aBBBEETEue (a2 . cnt  ^  I); 

tnii  ,aBBaEETEue(a2.  iaSEateCRcqSelectad”) )  ) 

a2.coiBpleteG0H|E]  ; 

iKi9.aBBaEtrEue(a2.iaStaie  ("Irror'')  )  i 
tnii .aBBaEEFalae (a2 . IsSucceBB  U ) ; 


Listing  5  Test  Case  3  for  AssertiDn  2 


Tbe&e  three  scenarios  were  similar  to  the  test  scenarios  of  the  first  Statechait 
assertion.  Hcre>  test  case  1  represented  a  typical  “happy"  scenario  where  all  requiremenis 
arc  adhered  to;  that  is,  for  every  requirement,  at  least  two  alicniatives  were  considered. 
For  the  second  test  case,  although  there  were  two  strategics,  they  belonged  to  two 
separate  requirements.  Upon  observirg  the  con^lcte  comptefeGQM  event,  the  Statechait 
assertion  cxiLs  the  ReqSclecied  stale.  The  data  strucnire  ent  that  ke^s  sack  of  the  number 
of  strategies  selected  will  note  that  in  this  case  only  one  strategy  was  selected  for  each 
requrrement,  and  they  will  transit  to  the  Error  stale.  For  the  third  case,  it  iii 
straightforward  as  only  one  strategy  was  selected  before  aUcmpling  to  exit  the 
RcqSeleclt'd  state.  Similarly,  the  assertion  failed  as  expected  and  the  Error  state  was 
entered. 

D.  APPLICATION  OF  THE  STATECHART  ASSERTION  FOR  THE 
RUNTIME  MONITORING  OF  THE  ACTUAL  ACQUISITION  PROCESS 


The  Statcchart  assertions,  which  were  developed  for  the  conccpt*rcfincmcnl  phase, 
can  be  af^lied  to  the  monitoring  of  the  actions  taken  by  the  cooccpt*retinemcnl  team  To 
achieve  this,  a  protocol  would  be  established  bebA'cen  the  PM  and  the  concepUrefinemenl 
team  The  protocol  would  describe  the  events  of  interest,  such  as  those  described  in  Tabic 
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1.  as  well  as  ihe  procedure  ro  follow  for  ihe  concept^refinemeni  learn  ro  nollfy  the  PM  of 
the  occurrence  of  these  events,  which  arc  hmc*stamped  and  recorded  in  an  event  log. 
Whenever  the  log  file  is  updated,  the  limc*stamped  event  sequences  in  Ihe  log  either  are 
translated  manually,  or  using  an  automated  tool,  into  JUnil  lest  scenarios  like  those 
shown  in  Listings  1*5  and  run  against  the  executable  assertions.  The  PM  will  be  notified 
whenever  an  event  sequence  violates  any  of  the  Statcchart  assertions. 
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V.  CONCLtSlON 


A.  SUMMARY 

In  this  thesis,  the  author  addressed  the  challenge  to  formally  specify  and  monitor 
the  workflow  for  the  process  to  acquire  a  complex  system  The  first  step  in  the  af^roach 
is  to  apply  the  GQM  method  to  identify  the  program  goals  and  develop  metrics  and  DRs 
to  measure  the  degree  of  success  in  accomplishing  the  goals  The  next  step  is  to  build 
activity  diagrams  to  model  the  workflow  process.  The  artifacts  generated  in  these  steps 
are  used  to  guide  the  formal  specification  of  the  workflow.  In  this  thesis,  the  author  u^ed 
Staiechart  assertions  as  the  formalism  and  the  StateRover  tool  to  demon.sirate  the 
technical  feasibility  of  validating  the  workflow  specification  and  the  tools  used  in 
conducting  automated  routine  execution  monitoring. 

The  acquisition  process  for  complex  systems  will  continue  to  evolve  and  reform. 
What  remains  invariant  is  the  need  for  assurance  that  the  process  is  faithfully  executed.  In 
this  regard,  the  author  modeled  a  part  of  an  acquisition  process  to  illustrate  that  program 
requirements  can  be  speeified  and  validated  using  Statechart  assertions.  It  is 
aeknowledgcd  that  the  Statechart  assertions  and  the  test  cases  presented  here  are  very 
simple  and  could  be  conducted  through  manual  examination  of  requirements  and  design 
artifacts  However,  this  simple  example  did  demonstrate  the  use  of  Statechart  assertion  as 
a  means  of  enforcing  process  requirements  The  test  cases  also  demonstrates  the  checking 
of  decision*making  and  the  adherence  of  requirements  In  the  context  of  the  modeled 
process. 

While  the  Statechart  assertions  were  not  applied  to  other  phases  of  the  acquisition 

process,  the  technique  could  still  be  valid  in  assuring  the  process  requirements  in  other 

phases.  In  addition,  this  technique  can  also  aid  developers  in  demonstrating  that  their 

software  satisfies  the  requirements  (functional  and  nonfunctional),  and  effectively  locates 

and  explains  the  cause  of  errors  in  faulty  design  and  development  In  most  complex 

systems,  for  example,  building  an  air  defense  system  correctly  that  meets  the  operational 

needs  of  the  battalion,  a  manual  V&V  technique  is  almost  inadequate.  Most  of  it  is  due  to 
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ihe  fine  level  of  detail  during  njniime  thal  makes  human  inlervenllon  almost  impractical 
Fortunately,  the  utilization  of  the  Statechait  assemons  and  the  SrateRover  tool  provided 
valuable  runtime  validation  of  behavioral  requirements  This  allows  the  stakeholders  to 
capture  the  tormal  requirements  to  ensure  that  the  developer's  cognitive  understanding  of 
the  requirements  matches  the  formal  specifications. 

B.  FUTURE  WORK 

More  work  is  needed  to  explore  the  application  of  the  approach  to  the  rest  of  the 
acquisition  process 

1 .  The  Statecharl  assertions  and  the  JUnit  test  case  scenarios  developed  using 
the  StateRover  can  be  extended  beyond  the  concept*rctincmcnt  phase  to 
the  other  phases  of  the  acquisition  process.  The  test  scenarios  can  be 
integrated  to  form  a  complete  system^acquisition  model.  Once  completed, 
it  is  necessary  to  include  a  suite  of  validation  test  scenarios  to  ensure  the 
correctness  of  the  formal  Statechart  assertions. 

2.  While  GQM  can  facilitate  requirements  analysis  and  guide  the 
development  of  metrics  to  measure  the  degree  to  which  program  goals  are 
met.  it  18  important  to  develop  the  complete  ^mework  to  use  the  metrics 
to  drive  the  f61low*on  efforts  and  measure  the  success  of  the  acquisition 
program. 

3.  The  process  of  creating  Statechart  assertions  from  the  activity  diagram  can 
be  rather  haphazard  because  it  relies  on  the  expertise  of  the  modeler.  One 
way  to  make  the  approach  more  structured  and  systematic  Is  to  develop 
ten^latcs  and  patterns  for  Identifying  natural  language  requirements  from 
the  UML  activity  dlagrams.^^  Another  way  to  reduce  the  burden  on  the 
modelers  and  to  in^rove  the  reliability  and  assurance  of  the  assertions  Is 
to  develop  libraries  of  prc*iestcd  generic  Statechart  assertions 
accompanied  by  scenario'based  test  cases.^^  Templates  and  reusable 
assertions  for  the  acquisition  process  need  to  be  developed  It  may  be 
possible  to  improve  the  quality  and  efficiency  of  an  acquisition  by 
equipping  developers  with  a  library  of  templates  and  generic  assertions. 

4.  The  author  propo.scd  to  use  runtime  execution  monitoring  to  assure  the 
proper  and  nmely  execuhon  of  the  acquisition  process.  The  artifact 
involved  was  the  log  file  containing  the  time-stamped  events  reported  by 

D.  Dnjsjisky,  “From  UML  Acdviiy  DiagKuns  to  Spccificaban  Requiremenis,'*  P/oc  IEEE 
ImemauoRal  CoRferertce  on  ofSy^iem^Engirieeting  (SoSE  W;.  June  2-4, 2008, 1-5. 

^  D.  Druunsky,  I.  B.  Michael,  T  W  Otani  and  M.  Shirty,  "Validating  UML  Statechan-Based 
Assertions  Libranes  for  Improved  Rdiabilily  and  Assurance,"  Pnc.  2*'  tntemalionat  Conference  on 
Secure  System  Ime^uon  and  keliahthty  fmprovement  (SElfU  tW),  July  14-17.  2008. 47-51 
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the  project  members.  It  was  noted  that  (o  execute  the  test  scenarios,  the  log 
tile  had  to  be  translated  to  the  JUnit  test  eases.  Manual  translation  is  not 
only  inefTieient  (as  the  author  ean  expect  a  lot  of  such  translations  in  an 
actual  acquisition  environment),  but  it  also  relies  heavily  on  expertise 
intervention  in  preparing  the  test  cases  The  resultant  JUnit  test  cases  also 
need  to  be  tested  to  ensure  that  they  are  working  correctly.  As  such,  a  one* 
time  developmental  cfTorl  should  be  invested  in  to  design  and  implement  a 
tool  to  automate  the  collection  of  the  nme^tamped  events,  the  translation 
of  the  event  log  into  a  JUmt  lest  case,  and  the  execution  of  JUnit  test 
against  the  Statechart  assertions.  A  report  could  subsequently  be  generated 
to  inform  the  PM  on  the  progress  of  the  processes  so  as  to  provide  runtime 
monitoring  and  recovery  from  assertion  failures.  The  only  thing  requiring 
manual  input  would  be  given  by  the  team  members  upon  the  completion 
of  their  tasks.  One  goal  the  develt^er  should  seek  is  to  automate  the 
processes  of  organizing,  renieving  information  from,  and  interfacing  the 
libraries  as  much  as  possible  in  an  attempt  to  reduce  errors. 
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